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Simulation  systems  are  being  increasingly  used  as  a  cheaper  alternative  to 
field  training,  and  as  the  Services  put  such  systems  into  place,  acquisition 
managers  must  add  new  methods  to  the  traditional  technical  performance 
measures  to  assess  the  effectiveness  of  these  training  systems. 


Today  each  Service  has  acquisition 
programs  under  way  to  provide  dis¬ 
tributed- simulation  systems  for  the 
collective  training  needs  of  military 
organizational  units.  Because  budgets  for 
collective  training  are  tight,  one  common 
objective  is  to  maintain  or  raise  unit  per¬ 
formance  by  acquiring  comparatively  less 
expensive  distributed-simulation  training 
systems  to  lessen  the  need  for  more 
expensive  field  training. 

Trading  field  training  for  distributed- 
simulation  training  systems  puts  pressure 
on  the  acquisition  community  to  ensure 
that  the  acquired  systems  are  successfully 
fielded  and  achieve  technical  performance 
objectives.  Assessing  those  objectives 
may  require  newly  developed  measures 
of  performance  that  mean  the  same  to 


the  acquisition,  supporting,  and  using 
communities. 

The  research  detailed  here  examines 
technical  performance  measures  for 
distributed-simulation  training  system 
acquisitions  used  for  collective  training 
of  military  units.  We  discuss  the  impor¬ 
tance  of  these  systems  to  the  acquisition 
community,  using  the  Army’s  Close  Com¬ 
bat  Tactical  Trainer  (CCTT)  as  an  example. 
We  identify  potentially  relevant  technical 
performance  measures.  And  finally,  we 
analyze  the  applicability  of  the  identified 
technical  performance  measures  during  an 
actual  distributed-simulation  training  sys¬ 
tem  exercise.  Findings  are  generalized  to 
other  such  systems  used  for  collective 
training  of  military  organizational  units. 
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The  Close  Com  bat  Tacti  cal  T rai  n er 

Historically,  to  avoid  poor  unit  perfor¬ 
mance  in  combat,  military  units  have 
focused  on  training  unit  tasks  through 
field  exercises.  With  declining  budgets, 
the  Services  are  acquiring  distributed- 
simulation  systems  that  are  perceived  to 
be  able  to  train  units  more  cheaply  than 
do  field  exercises. 

All  the  Services  and  many  joint  orga¬ 
nizations  are  acquiring  these  systems. 
Under  the  direction  of  the  Under  Secretary 
of  Defense  for  Acquisition,  the  Defense 

Science  Board 
(Foster,  1993) 
began  to  define 
some  of  the 
terms  appli¬ 
cable  to  these 
systems.  The 
Board  used  the 
term  “distrib¬ 
uted”  to  refer 
to  a  “shared 
battlefield  entered  from  geographically 
separated  sites  via  communication  net¬ 
works.”  The  Board  also  defines  simula¬ 
tion  as  a  “mix  and  match  of ...  simulation 
methods.”  Since  then  the  Defense  Mod¬ 
eling  and  Simulation  Office  (1995)  has 
promoted  the  simulation  “mix  and  match” 
concept  through  “a  general  purpose 
architecture  for  simulation  reuse  and 
interoperability”  called  the  high-level 
architecture  (HLA). 

The  Defense  Modeling  and  Simulation 
Office  is  developing  an  HLA  that  will 
enable  multiple  simulation  federations 
(groups  of  simulations)  to  exist  within  and 
between  all  the  Services,  joint  commands, 
and  others.  Many  future  distributed-simu¬ 
lation  federations  and  systems  are  planned 


for  training  use  (Hammond  and  Edwards, 
1998). 

The  Advanced  Research  Projects 
Agency  has  composed  a  joint  simulation 
federation  used  for  collective  training — 
containing  air,  naval,  and  army  simulated 
elements  within  the  Synthetic  Theater  of 
War  initiative  (Meier,  1999).  Another  sys¬ 
tem,  the  “distributed  mission  trainer” 
(DMT),  is  a  priority  for  the  Air  Combat 
Command  (Hawley,  1998).  When  fielded 
in  1999,  DMT  will  add  integrated  and  dis¬ 
tributed  manner  simulator  systems  at 
Eglin,  Eangley,  Shaw,  and  Tinker  Air 
Eorce  Bases  (AEBs)  to  simulation  systems 
already  in  the  Air  Eorce  (Kuhn,  1998).  The 
integration  will  provide  a  complete  spec¬ 
trum  of  aircraft  and  facilities  for  Air  Eorce 
unit  training  and  mission  rehearsal. 

An  illustrative  example  of  a  distributed- 
simulation  training  system  acquisition  is 
the  Army’s  CCTT,  currently  being  fielded. 
As  do  field  training  exercises,  the  CCTT 
“will  train  Armor,  Cavalry,  and  Mecha¬ 
nized  Infantry  Platoons  through  Battal¬ 
ion/Task  Eorce  on  their  doctrinal  Mission 
Training  Plan  collective  tasks” 
(Hammond  and  Edwards,  1998). 

But  unlike  field  training  exercises,  “the 
CCTT-system  ...  consists  of  networked 
vehicle  simulator  manned-modules  ...  in 
combination  with  Semi-Automated 
Eorces,  Combat  Support  workstations, 
computer  networks  and  protocols,  and 
After-Action  Review  systems” 
(Hammond  and  Edwards,  1998).  Actual 
military  systems  like  tanks  are  not  used 
in  the  CCTT  distributed-simulation 
training  system.  The  CCTT  may  be 
considered  a  distributed,  synthetic  battle¬ 
field  with  various  simulators  that  enable 
virtual  and  other  synthetic  players  to 
interact  in  simulated  battles. 


"Historically,  to 
avoid  poor  unit 
performance  in 
combat,  military 
units  have  focused 
on  training  unit 
tasks  through 
field  exercises." 
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As  in  field  training  exercises,  senior 
evaluators  and  unit  leaders  discuss  unit 
task  and  mission  accomplishments  and 
failures  with  unit  members  after  the  CCTT 
training.  This  forum  is  referred  to  as  an 
“after- action-review.”  After-action- 
reviews  also  provide  instruction  on 
process  improvements  that  are  aimed  at 
improving  overall  unit  performance. 

A  simple  analogy  for  this  review 
session  might  be  the  discussion  that  a  high 
school  basketball  team  coach  has  with  his 
team  immediately  after  a  team  scrimmage. 
The  emphasis  in  practice  is  on  processes 
like  individual  dribbling  and  shooting,  and 
team  plays  like  setting  up  a  clear  three- 
point  shot  or  a  fast  break.  The  coach  does 
not  focus  so  much  on  the  score  (outcome 
measure)  run  up  against  the  scrimmage 
squad,  but  rather  uses  those  failures  and 
successes  as  points  to  correct  specific  task 
errors  or  reinforce  successes.  In  combi¬ 
nation  with  personal  and  other  team  tasks 
and  plays,  their  ability  to  perform  these 
tasks  affects  their  ability  to  put  points  on 
the  scoreboard  in  the  real  game. 

Unlike  most  field  training  exercises, 
with  the  exception  of  some  live  simulation 
sites,  sophisticated  after-action-review 
systems  permit  replay  of  portions  of  the 
unit  actions  that  occurred  during  a  CCTT 
exercise.  These  after-action  review  sys¬ 
tems  enhance  unit  discussion  and  further 
enable  unit  performance  improvements. 

As  in  field  training,  U.S.  Army  mecha¬ 
nized  and  armored  units  (platoons, 
companies,  and  battalions)  use  mission 
training  plans  in  the  CCTT.  These  plans 
identify  general  and  specific  tasks  with 
conditions  and  standards  for  measuring 
unit  performance  against  these  missions. 

Units  tend  to  build  ever-higher  levels 
of  competence  through  exposure  to  ever 


greater  challenges  in  training  (CCTT, 
1998).  In  the  dynamics  of  human  and  unit 
growth,  the  learning  environment  evolves. 
From  learning 
basic  unit  tasks, 
moving  on  to 
learn  advanced 
unit  tasks,  rein¬ 
forcement  of 
previously 
learned  tasks, 
and,  finally,  in¬ 
tegration  of 
various  combinations  of  tasks  (typically 
a  mission  or  set  of  missions),  individuals 
learn  through  some  combination  of 
instruction,  discussion,  and  exercises. 

Just  as  in  a  field  or  live  simulation 
exercise,  the  distributed- simulation  train¬ 
ing  exercise  integrates  tasks  in  the  form 
of  unit  mission  scenarios.  The  training  goal 
is  to  learn  and  perfect  unit  integrated  pro¬ 
cesses  like  unit  tactics,  techniques,  and 
procedures  that  are  transferable  to  many 
different  missions.  The  focus  is  typically 
not  exclusively  about  the  resulting  out¬ 
come  for  a  particular  mission.  Similar  to 
the  basketball  scrimmage  example,  the 
emphasis  is  not  on  the  outcome  of  the  scrim¬ 
mage,  but  on  the  processes  that  can  put 
points  on  the  board  during  the  real  game. 

Relationship  of  Systems  to 
theAcouisition  Manager 


As  these  distributed-simulation  train¬ 
ing  systems  emerge  and  move  toward 
fielding,  the  need  becomes  apparent  for 
metrics  to  help  communicate  meaning 
between  dissimilar  communities  and  to 
evaluate  them  appropriately.  As  an 
example  of  the  importance  of  metrics. 


"These  after-action 
review  systems 
enhance  unit 
discussion  and 
further  enable 
unit  performance 
Improvements." 
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consider  the  Williams  and  Keaton  (1998) 
comprehensive  evaluation  of  the  CCTT 
1997-1998  initial  operational  test  (lOT) 
and  1997  limited  user  test  (LUT)  con¬ 
ducted  hy  the  Test  and  Experimentation 
Command  (TEXCOM). 

During  lOT  fixed-site  simulator  train¬ 
ing  in  the  CCTT,  Williams  and  Keaton 
report  (based  on  their  aggregate  task  mea¬ 
sures)  only  a  “range  of  modest  to 
insignificant  gains  observed  during  the 
CCTT  training.”  Specifically,  “simulator 
training  during  the  third  through  seventh 
weeks  of  the  lOT  indicates  that  few  per¬ 
formance  gains  were  achieved  by  the  units 
undergoing  training.” 

Despite  this  recorded  lack  of  perfor¬ 
mance  gains  in  the  CCTT,  Williams  and 
Keaton  report  that  “At  the  aggregate  level 
across  all  subtasks,  the  CCTT  [-trained] 
units  performed  significantly  better  at 
NTC”  (National  Training  Center  field 


exercise)  than  other  baseline  task  forces 
(Eigure  1)  (Williams  and  Keaton,  1998). 
Specifically,  aggregating  company  team 
performance  within  each  observed  task 
force,  “Task  Eorce  4  [TE4],  the  CCTT  test 
unit,  clearly  outperformed  the  three 
baseline  task  forces.” 

There  may  be  many  different  explana¬ 
tions  for  these  starkly  different  observa¬ 
tions  of  performance.  One  would  hope 
that  CCTT  training  was  the  primary 
contributor  to  success  at  the  NTC.  But 
Williams  and  Keaton  conclude,  “The  lOT 
in- simulator  performance  data  was  insuf¬ 
ficient  to  demonstrate  a  linkage  between 
CCTT  training  and  performance  attained 
in  the  field.” 

One  alternative  conclusion  is  that 
additional  measures  and  measurement 
instruments,  or  a  different  approach,  are 
needed  to  capture  unit  performance 
improvement  that  may  have  actually 


Figure  1.  Pure  CCTT  Company  Teams  (TF4)  versus 
All  Baseline  Company  Teams  (TF  1,2,3) 
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occurred  while  training  in  distributed- 
simulation  systems.  Supporting  this  alter¬ 
native  conclusion  is  the  fact  that  these  sys¬ 
tems  focus  on  improving  the  processes 
that  make  a  unit  function  and  not  so  much 
on  the  outcome  of  a  particular  training 
scenario. 

Since  distributed-simulation  training 
systems  are  acquisitions,  both  the  acqui¬ 
sition  community  and  the  training  com¬ 
munity  could  benefit  from  a  shared  lexi¬ 
con  of  technical  performance  measures  to 
provide  a  clearer  indication  of  overall 
potential  to  achieve  ultimate  technical 
objectives. 

In  1998  Maj  Kenneth  Delano  published 
a  survey  of  program  managers  showing 
that  meeting  “technical  performance 
objectives”  is  ranked  first  as  an  indicator 
of  program  success.  His  survey  also 
revealed  that  program  managers  ranked 
their  own  “ability  to  communicate”  as  the 
most  important  factor  in  program  success 
(Delano,  1998).  (The  article  did  not  define 
either  of  these  terms  more  definitively.) 

Aiding  both  evaluation  and  communi¬ 
cations,  The  Systems  Acquisition 
Manager’s  Guide  for  the  Use  of  Models 
and  Simulations,  published  by  the 
Defense  Systems  Management  College 
(Piplani,  Mercer,  and  Roop,  1994), 
identifies  numerous  outcome-oriented, 
technical  performance  measures  for  use 
by  acquisition  managers  of  combat 
systems. 

By  contrast,  reports  on  technical  per¬ 
formance  measures  for  unit  collective 
training  systems  are  scant  if  at  all  present 
in  the  acquisition  literature.  Further, 
technical  performance  measures  for  evalu¬ 
ation  of  individual  training  systems  have 
traditionally  been  submerged  within  the 
related  combat  system  acquisition. 


Typically,  training  systems  were  justified 
as  trainers  for  a  specific  aircraft,  weapon 
system,  etc.  Consequently,  technical 
performance  measures  for  individual  and 
crew-training  systems  have  been  system 
specific  and  oriented  to  system 
performance. 

Teamwork  and  Task  Performance 


The  most  applicable  traditional  tech¬ 
nical  measure  identified  in  the  Piplani  et 
al.  (1994)  publication  is  an  aggregate  out¬ 
come  measure  referred  to  as  loss  exchange 
ratio  (LER).  The  LER  can  be  used  to  judge 
individual  or  crew  performance  improve¬ 
ments.  The  EER  is  an  outcome  measure 
that  compares  enemy  losses  to  friendly 
losses.  Using  an 
air  warfare  anal¬ 
ogy,  a  loss  ex¬ 
change  ratio 
might  compare 
the  number  of 
enemy  aircraft 
shot  down  to 
the  number  of 
friendly  aircraft 
shot  down.  The 
more  enemy 
aircraft  shot  down  for  every  friendly  air¬ 
craft  shot  down,  the  better  your  system. 
A  difficulty  in  this  approach  is  that  it  is 
limited  in  scope  to  comparative  systems/ 
units  and,  in  a  peacetime  environment 
without  actual  adversaries,  the  EER 
becomes  suspect. 

Eurther,  Johnston,  Smith-Jentsch,  and 
Cannon-Bowers  (1997),  Smith-Jentsch, 
Johnston,  and  Payne  (in  press-a)  and 
Brannick,  Prince,  Prince,  and  Salas  (1995) 
indicate  that  “free  play”  training  exercises 


"  ...reports  on 
technical  perfor¬ 
mance  measures 
for  unit  col lective 
training  systems 
are  scant  if  at  all 
present  in  the 
acquisition 
literature.." 
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produce  inconsistent  outcomes  in  the  LER 
when  measuring  unit  performance  change 
from  training  period  to  training  period, 
whereas  the  alternative  to  “free  play” — a 
structured  exercise — was  expensive  to 
huild  and  maintain. 

As  possible  supplemental  measures, 
Glickman  et  al.  (1987),  McIntyre  and 
Salas  (1995),  and  others  discussed  the 
influence  of  teamwork — a  collection  of 
critical  behaviors  and  interpersonal 
skills — on  unit  or  collective  task  perfor¬ 
mance.  These  two  technical  performance 
measures — teamwork  and  unit  task  per¬ 
formance — are  not  widely  discussed  in  the 
system  acquisition  literature.  As  measures 
they  represent  analysis  of  the  process  as 
opposed  to  the  aggregate  outcome  of  those 

processes. 

Johnston  et 
al.  (1997)  re¬ 
fined  these 
teamwork  di¬ 
mensions  and 
in  a  second  re¬ 
view,  Smith- 
Jentsch  et  al. 
(in  press-a  and 
-b)  refined  the 
four  teamwork  dimensions — discussed 
below — into  more  reliable  and  indepen¬ 
dent  dimensions  containing  sets  of  spe¬ 
cific  interpersonal  behaviors.  Qualitative 
assessment  for  each  dimension  and 
behavior  can  be  done  using  behaviorally 
anchored  rating  scales  (Johnston  et  al., 
1997). 

The  use  of  both  process  measures  to 
supplement  outcome  measures  allows  for 
a  more  complete  assessment  of  system 
contribution  (Brannick  et  al.,  1995; 
Johnston  et  al.,  1997;  Smith-Jentsch  et  al., 
in  press-a;  Stout,  Cannon-Bowers,  and 


Salas,  1997).  This  is  significantly  differ¬ 
ent  from  the  more  familiar  and  traditional 
emphasis  on  outcome  measures  identified 
by  Piplani  et  al.  (1994). 

Collective  Training  in  a  Distributed- 
Simulation  Training  System 


So  what  research  could  better  illumi¬ 
nate  the  contribution  of  process  measures 
as  a  supplement  to  outcome  measures  with 
respect  to  the  discussion  and  evaluation 
of  distributed  simulations  used  for 
training? 

During  our  research  we  observed  col¬ 
lective  training  in  the  CCTT  distributed- 
simulation  training  system  to  gain  insight 
into  what  teamwork  and  task  performance 
measures  might  provide. 

Johnson  and  Noble  (1994)  indicate  that 
distributed  interactive  simulation  has  the 
potential  to  effectively  train  the  follow¬ 
ing  primary  tasks:  command,  control  and 
communications  (C3);  maneuver  and 
navigation;  teamwork,  and  leadership. 

For  this  study,  the  research  team  inves¬ 
tigated  measures  for  two  of  these  tasks — 
teamwork  and  C3  task  performance — by 
observing  the  normal  training  of  two 
active-duty  U.S.  Army  battalion  task 
forces  within  the  CCTT  facility  at  Fort 
Hood,  TX.  Each  battalion  task  force 
reported  to  the  CCTT  facility  to  conduct 
training.  The  battalion  task  force  received 
familiarization  training  on  the  CCTT  and 
then  practiced  operating  and  maneuver¬ 
ing  manned  module  vehicles  and  units 
within  the  CCTT. 

In  the  recorded  training  exercise,  the 
task  force  received  a  “movement  to 
contact”  mission  and  entered  its  tactical 
operations  planning  process.  A  tactical 


"The  use  of  both 
process  measures 
to  supplement 
outcome  measures 
allows  for  a  more 
complete  assess¬ 
ment  of  system 
contri  bution...." 
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plan  was  devised,  rehearsed,  and  then 
executed  in  the  CCTT. 

Next,  each  battalion  task  force  con¬ 
ducted  an  after-action-review  feedback 
session  on  unit  tactical  performance  at 
both  the  company  level  and  the  battalion 
level.  After  that,  the  units  repeated  the 
movement-to-contact  mission.  Upon 
completion  of  the  second  trial,  another 
feedback  session  was  conducted  to  assess 
tactical  performance. 

Participants  in  the  study  served  in  duty 
positions  that  included  company  com¬ 
mander,  executive  officer,  and  platoon 
leaders.  For  the  purpose  of  this  study,  this 
team  of  leaders  was  referred  to  as  the  tac¬ 
tical  command  and  control  team,  since 
these  individuals  provide  the  leadership 
to  command  and  control  their  units  while 
executing  their  mission. 

For  control  purposes  the  same  move¬ 
ment-to-contact  scenario  was  used 
between  the  first  and  second  simulation 
run.  The  selected  tasks  to  be  performed 
were  identical  between  simulation  runs. 
The  scenario  in  each  run  presented  the 
same  mission,  enemy  force,  terrain,  time 
frame,  environmental  conditions,  and 
semiautomated  entities’  coded  behavior. 
The  opposing  force  consisted  entirely  of 
semiautomated  forces  under  the  control 
of  an  experienced  operator. 

The  semiautomated-force  operators 
used  their  “free  play”  prerogative  in  the 
second  run.  Specifically,  units  typically 
train  against  a  lesser  able  opposing  force 
in  their  initial  training.  In  accordance  with 
the  learning  objectives  of  the  command¬ 
ing  officer,  the  semiautomated-force 
operators  typically  increase  the  degree  of 
difficulty  by  increasing  the  quality  of 
semiautomated-force  tactical  operations 
in  subsequent  runs  (CCTT,  1998). 


As  indicated  above,  this  common 
training  approach  with  increasing  diffi¬ 
culty  in  subsequent  training  exercises  was 
suspected  to  influence  LER  relation¬ 
ships.  Since  the  research  was  aimed  at 
supplementing  the  LER  as  a  technical 
measure,  the  research  team  collected  LER 
data. 

We  wanted  to  evaluate  the  use  of 
process  measures  in  light  of  aggregate 
outcome  mea¬ 
sures.  To  facili¬ 
tate  the  evalua¬ 
tion,  we  used  an 
event-based  ap¬ 
proach  to  focus 
on  teamwork 
dimensions  and 
unit  task  perfor¬ 
mance  of  each 
company’s  tactical  command  and  control 
team  during  each  movement-to-contact 
mission. 

Each  event  contains  a  unique  tactical 
situation  that  requires  team  members  to 
coordinate  and  exchange  information  at 
each  step  in  order  to  assess  the  situa¬ 
tion,  make  the  appropriate  decisions,  and 
execute  the  correct  actions.  We  selected 
three  specific  events  that  were  likely  to 
require  the  execution  of  team  behaviors. 
Hence  the  mission  was  broken  up  into 
three  events.  The  events  selected  were: 

•  perform  tactical  movement  (17-2- 
0301); 

•  perform  actions  on  contact  (17-2- 
0304);  and 

•  perform  an  attack  by  fire  (7 1  -2-03 11) 
(Department  of  the  Army,  1988). 


"  Participants  in 
the  study  served  in 
duty  positions  that 
included  company 
commander,  execu¬ 
tive  officer,  and 
platoon  leaders." 
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Each  event  has  task  steps  (processes) 
or  suhtasks,  some  which  have  critical 
suhtasks  associated  with  them.  Success 
or  failure  of  each  task  step  was  recorded. 

A  set  of  teamwork  observation  forms 
(measurement  instruments)  adapted  from 
methodology  used  hy  both  Johnston  et  al. 
(1997)  and  Smith-Jentsch  et  al.  (1996a) 
were  applied  for  each  of  four  teamwork 
dimensions:  communication,  information 
exchange,  team  initiative  and  leadership, 
and  supporting  behaviors  (team  initiative/ 
leadership.  Figure  2). 


Team  behaviors  were  categorized  into 
effective  and  ineffective  team  behavior. 
Team  behavior  quality  ratings  were 
assessed  three  ways:  the  ratio  of  effective 
to  ineffective  team  behaviors,  the  impact 
or  severity  of  specific  team  behaviors,  and 
an  overall  subject-matter  expert  rating  of 
team  behaviors.  Quality  ratings  assessed 
the  four  teamwork  dimensions  and 
specific  team  behaviors  (Table  1)  using  a 
1  to  5  Likert  scale. 


Team  Initiative/Leadership 


Event  One.  Tactical  movement  en  route  to  enemy  contact:  Event  One  begins  at  start  of  scenario  and 
iasts  untii  contact  is  made  with  an  enemy  force. 

Task:  PERFORM  Tacticai  Movement  (1 7-2-0301 )  Ref:  FM  71  -1 

Unit: _  Simuiation  Run: _ 


Ciear  and  appropriate  guidance  provided  to 

team  when  needed. 

1  2  3 

1  1  1 

4  5 

1  1 

Guidance  is 

Clear  and 

unclear  or  never 

appropriate 

stated. 

guidance 

always 

stated. 

Ciear  and  appropriate  guidance  provided  to 

team  when  needed. 

1  2  3 

1  1  1 

4  5 

1  1 

Priorites 

Clear  and 

unclear  or 

appropriate 

never  stated. 

priorities 

always 

stated. 

Ciear  and  appropriate  guidance  provided  to 

team  when  needed. 

1  2  3 

1  1  1 

4  5 

1  1 

Reform 

Reform 

inappropriate 

appropriate  and 

or  never  stated. 

always  stated. 

Team  Initiative/Leadership  Frequency 

•  Provide  guidance  or  0 
suggestions  (Effective) 

1-5 

6-10 

>10 _ 

•  Provide  guidance  or  0 
suggestion  (Ineffective) 

1-5 

6-10 

>10 _ 

•  States  clear  team/  0 

Individual  priorities 
(Effective) 

1-5 

6-10 

>10 _ 

•  States  priorities  0 

(Ineffective) 

1-5 

6-10 

>10 _ 

•  Refocus  team  lAW  0 

situation  (Appropriate) 

1-5 

6-10 

>10 _ 

•  Refocus  team  lAW  0 

situation  (Inappropriate) 

1-5 

6-10 

>10 _ 

Figure  2.  Sample  Team  Observation  Worksheet 
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Table  1.  Teamwork  Dimensions  and  Team  Behaviors 


Teamwork  Dimensions 

Definitions  of  Team  Behaviors 

Information  exchange 
(Effective  behaviors,  1-4; 
ineffective  behaviors,  5-7) 

1.  Seeks  information  from  available  sources 

2.  Passes  information  to  the  appropriate  persons 

3.  Provides  accurate  “big  picture”  situation  update 

4.  Accurately  informs  higher  commander 

5.  Has  to  be  asked  for  information 

6.  Provides  inaccurate  situation  update 

7.  Inaccurately  informs  higher  commander 

Communication 
(Effective  behaviors,  1-3; 
ineffective  behaviors,  4-7) 

1.  Uses  proper  phraseology 

2.  Provides  complete  reports 

3.  Adequate  brevity;  avoids  excess  chatter 

4.  Uses  improper  phraseology 

5.  Provides  incomplete  reports 

6.  Uses  excessive  chatter 

7.  Communications  are  inaudible  or  garbled 

Team  initiative/leadership 
(Effective  behaviors,  1-3; 
ineffective  behaviors,  4-6) 

1 .  Provides  effective  guidance  or  suggestions  to  team 
members 

2.  States  clear  team  and  individual  priorities 

3.  Appropriately  refocuses  team  in  accordance  with 
situation 

4.  Provides  ineffective  or  unclear  guidance  or  sugges¬ 
tions  to  team  members 

5.  States  ineffective  or  unclear  team  and  individual 
priorities 

6.  Inappropriately  refocuses  team  in  accordance  with 
situation 

Supporting  behavior 
(Effective  behaviors,  1-4; 
ineffective  behaviors,  5-6) 

1 .  Corrects  team  errors 

2.  Requests  backup  or  assistance  when  needed 

3.  Provides  backup  or  assistance  when  needed 

4.  Provides  constructive  feedback 

5.  Fails  to  correct  team  errors 

6.  Provides  or  uses  nonconstructive  feedback 

In  order  to  avoid  inconsistency  of 
assessment  between  multiple  observers, 
one  evaluator  was  trained  and  validated 
at  100%  proficiency  in  identification  and 
classification  of  teamwork  dimensions 
and  respective  behaviors  by  using  tbe 
“team  dimensional  training”  computer- 
based-instructional  software  program 
(Smith-Jentscb,  Zeisig,  Acton,  and 
McPherson,  in  press-b).  The  same 


observer  assessed  team  behavior  quality 
ratings  and  team  task  performance  for  all 
teams. 

Results  AND  Analysis 


After  training  was  completed,  the 
CCTT  after-action-review  tapes  were  ana¬ 
lyzed  to  observe,  categorize,  and  record 
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observations  of  teamwork  and  task  per¬ 
formance.  Eight  company-level  move- 
ment-to-contact  mission  scenarios  were 
evaluated.  All  radio  communications  and 
team  or  unit  actions  were  observed  and 
monitored  separately  for  each  tactical 
command  and  control  team. 

Task  performance  was  assessed  for 
each  task  event  based  on  the  U.S.  Army 
Mission  Training  Plan  for  Tank  and 
Mechanized  Infantry  Company  and  Com¬ 
pany  Team  mentioned  earlier.  Teamwork 
dimensions  and  team  behaviors  were  ana¬ 
lyzed  for  indication  of  improvement.  In 
addition,  the  traditional  loss  exchange 
ratio  measure  was  evaluated. 

As  a  means  of  analyzing  C3  task  per¬ 
formance,  a  series  of  matched  pairs,  one- 
tailed  Mests  compared  the  difference  in 
critical  task  and  subtask  success  between 
simulation  run  No.  1  and  run  No.  2. 
Matched  pairs,  one-tailed  t-tests  compared 
loss  exchange  ratios  differences  between 


runs  but  due  to  the  nature  of  the  selected 
tasks,  not  all  tasks  involved  a  LER.  Eor 
all  statistical  tests  a  significant  difference 
was  declared  if  the  probability  of  random 
occurrence  was  less  than  or  equal  to  0.05. 

G  Task  Performance  and 
LER  Results  AND  Analysis 

C3  task  performance  indicated  statisti¬ 
cally  significant  improvement  in  mission 
training  plan  critical  subtask  (p  =  .044) 
and  total  subtask  success  (p  =  .007)  as 
shown  in  Eigure  3.  Eor  the  EER,  from  a 
sample  of  eight  tasks  that  did  involve  the 
EER,  five  teams  had  an  increase  in  the 
EER,  two  teams  had  a  decrease  in  the 
EER,  and  one  team  had  no  change  in  the 
EER.  Statistically  the  EER  did  not  indi¬ 
cate  any  change  due  to  the  variability  in 
the  sample,  although  the  change  was 
relatively  large  as  Eigure  3  shows  (p  =  .57). 

Variations  in  task  difficulty  during  the 
second  run,  due  to  the  free  play  in  the 
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Change  in  Performance 

Figure  3.  Performance  Change  between  Run  No.  1  and  Run  No.  2 
based  on  Cited  Technical  Performance  Measure 


□  %  Critical  tasks  improved 

□  %  Subtasks  improved 
I  I  %  of  LER  change 
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simulation,  altered  some  tactical  issues 
(holding  of  key  [advantageous]  terrain, 
enemy  initiating  contact  from  a  hasty 
defense/attack-hy-fire  positions,  force 
ratio  of  attacking  to  defending  units,  etc.). 
As  suspected,  these  variations  in  task  dif¬ 
ficulty  between  the  two  runs  may  have 
influenced  or  confounded  the  outcomes 
of  the  LER  data. 

Teamwork  Quality  Ratings 

In  order  to  determine  if  training  in 
virtual  simulation  resulted  in  an  improve¬ 
ment  in  teamwork,  teamwork  quality 
ratings  were  assessed  for  each  run.  Qual¬ 
ity  ratings  between  the  first  and  second 
runs  were  found  to  have  improved  to  a 
statistically  significant  degree  for  all  team¬ 
work  dimensions.  Additionally,  improve¬ 
ments  in  quality  ratings  for  13  team 
behaviors  that  make  up  the  teamwork 
dimensions  were  found  to  be  significant. 

Conclusions 


Whether  in  athletic  competition  or  in 
combat,  quality  teamwork  demonstrates 
its  tremendous  value.  An  example  process 
or  task  worked  on  by  a  highly  skilled  team 
might  be  the  “no-look  pass”  between  bas¬ 
ketball  players  Michael  Jordan  and  Scot- 
tie  Pippen.  Their  teamwork  created  many 
national  championships. 

In  the  past  the  U.S.  Armed  Forces  have 
had  significant  opportunity  to  develop  ex¬ 
pertise  in  unit  teamwork  and  mission  task 
performance  during  training  involving 
field  operations  using  actual  equipment 
and  formations.  That  environment  has 
changed  significantly  due  to  a  number  of 
factors.  But  despite  the  change,  we  don’t 
want  to  become  the  Chicago  Bulls  of 


1999.  To  compensate,  our  armed  forces 
appear  ready  to  acquire  less  costly  dis¬ 
tributed-simulation  training  systems  in 
order  to  help  fill  the  gap  created  by 
reduced  field  training. 

These  findings  indicate  that  training  in 
distributed-simulation  systems  can  result 
in  statistically  significant  improvements 
in  teamwork,  C3  task  performance,  and, 
potentially,  the  loss  exchange  ratio. 
Specifically,  our  research  indicates  that 
distributed-simulation  training  systems 
can  help  fill  at  least  two  gaps — teamwork 
and  C3  task  performance.  Statistically 
significant  improvements  in  the  quality 
of  teamwork  were  shown  while  conduct¬ 
ing  training  in  a  distributed-simulation 
training  system.  Also,  C3  task  perfor¬ 
mance  was  found  to  significantly  improve 
between  training  sessions  as  shown  by 
increased  mission  training  plan  critical 
subtask  and  total  subtask  successes. 

Our  study  also  indicates  traditional 
measures  such  as  loss  exchange  ratios  do 
not  appear  to  be  appropriate  as  sole 
technical  measure  when  evaluating  the 
suitability  of  simulation  systems  used  for 
training.  We  observed  no  overall  statisti¬ 
cally  significant  change  in  task  perfor¬ 
mance  between 
simulation  runs 
as  measured  by 
the  LER.  As  of¬ 
ten  the  case  in 
training,  the 
LER  may  not 
be  a  credible  in¬ 
dicator  of  im¬ 
proved  proficiency  of  the  unit  as  the 
difficulty  of  the  opposing  force  fight 
might  increase  for  training  purposes  from 
run  No.  1  to  run  No.  2.  While  duplication 
of  the  same  scenario  and  difficulty  level 


"Whether  in 
athletic  competition 
or  in  combat, 
quality  teamwork 
demonstrates  its 
tremendous  value." 
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is  possible  in  training,  this  typically  only 
occurs  for  units  that  fail  that  level  and  need 
to  be  retrained. 

The  findings  provide  initial  indication 
that  process-oriented  measures  such  as 
teamwork  and  subtask  and  step  perfor¬ 
mance  are  viable  and  should  supplement 
the  acquisition  managers’  set  of  technical 
performance  measures  for  distributed- 
simulation  training  systems.  These  mea¬ 
sures  provide  the  acquisition  system 
manager  a  more  complete  assessment  of 
the  ability  of  a  prospective  distributed- 
simulation  training  system  than  loss 
exchange  ratio  would  by  itself.  Further, 
these  measures  are  intuitive  and  simple, 
helping  to  satisfy  the  challenge  of 
communications  as  well  as  evaluation. 

We  identify  our  measures,  approach, 
and  measurement  instruments,  which  may 
prove  useful  for  more  general  application 
to  other  distributed-simulation  acquisition 


involving  collective  training.  Further,  they 
appear  appropriate  not  only  for  U.S.  Army 
acquisitions  but  also  for  the  Air  Force, 
Navy,  and  Joint  organizations  in  light  of 
DMT  and  HLA  efforts.  These  findings 
imply  that  these  process-oriented  tech¬ 
nical  performance  measures  and  method¬ 
ologies  may  be  additional  tools  with 
which  astute  acquisition  manager  should 
be  familiar. 

Further  research  is  required  to  deter¬ 
mine  if  these  findings  can  be  confirmed 
with  larger  sample  sizes,  perhaps  over 
time  and  across  other  distributed-simula¬ 
tion  systems  used  for  collective  training. 
Further  research  may  address  the  appli¬ 
cation  of  these  findings  to  other  training 
audiences  within  distributed-simulation 
such  as  air  wings,  ship  command  and 
control,  higher  staffs,  and  other  types  of 
organizations. 
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